In industrial test and maintenance projects, test execution plans are important for performing test cycle in a time constrained manner with the objective of delivering the expected quality by effective utilization of resources. Regression testing is an important phase in software test Life cycle and contributes significantly in day-to-day operations in industrial software systems. Executing the test cases of the regression test suite repetitively for each change is a resource-and-time-intensive process. With ever reducing test cycle times, test teams face tremendous pressures to complete regression testing in time and budget, often compromising on quality.
Test teams distribute test cases among multiple resources to execute them in parallel.
To achieve this, it is important to have an execution plan that simplifies work load distribution and guarantees that expected outcomes of the tests do not change in spite of parallelism. In practice, these plans are often created based on the understanding and experiences of the testers. But due to various dynamic parameters such as regression suite sub setting for a particular change, varying team size, immediate release requirements and abends, the static plans often fail to provide effective guidance, and needs to be amended. The concept of parallel execution of test case has been explored in prior arts where the specified test cases have been executed in parallel. Some have also utilized cloud frameworks to achieve testing in parallel and distributed manner. However, all these approaches require the execution plan to be specified by the test designers.
There often exist dependencies across the test cases that determine the order of execution and are required to be preserved during the executions. Out-of-order execution of these test cases may lead to unexpected results. Analysis of failures to attribute the cause to, either a defect in the program or incorrect state of the system is time-consuming and defeats the purpose of parallelism. Dependencies are prevalent in scenario-based testing of database applications and GUI systems where dynamically created UI elements (e.g. web-pages, screens, etc) are tested in a particular sequence. Apart from the application functionalities and persistence data accesses in collaborative testing, test data reuse also creates dependencies across test cases. In all these cases, an execution plan must oblige to the dependencies and sequences, while exploiting the parallelism across the test cases.
An attempt has been made in the past to identify impact of test case execution sequences and partition test cases based on state preservation by resetting the states. However, all these approaches assume that on reported failure, state can be reset and all test cases can start from the initial state. State resets can be expensive and the assumption that all test cases start from same initial state may not hold true. The option of clustering test cases in of resource constraints has also been explored, but failed as it does not consider the dependence across the test cases.
An effective step towards addressing this problem is to dynamically generate a test execution plan that is derived using balanced partitioning and scheduling techniques and is based on the cost of executions of the test cases. The plan shall be capable of facilitating parallel execution of test cases within available resources, and considering the test case dependencies.